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Abstract 


Design  for  Six  Sigma  (DFSS)  is  an  industry  recognized  methodology  used  by  Raytheon’s  Inte¬ 
grated  Product  Development  System  to  predict,  manage,  and  improve  software -intensive  system 
performance,  producibility,  and  affordability.  The  Raytheon  Integrated  Defense  Systems  DFSS 
team  has  developed  and  implemented  numerous  leading-edge  improvement  and  optimization 
methodologies  resulting  in  significant  business  results.  These  achievements  have  been  recognized 
by  the  Software  Engineering  Institute  and  the  Institute  of  Electrical  and  Electronics  Engineers 
with  the  2016  Watts  Humphrey  Software  Process  Achievement  Award.  Best  practice  approaches 
used  by  the  team  are  shared  in  this  report,  including  the  generation  of  highly  effective  and  effi¬ 
cient  test  cases  using  Design  of  Experiments,  process  performance  modeling  and  analysis,  and 
cost  and  schedule  risk  analysis  using  Monte  Carlo  simulation. 
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Enabling  Improved  Software  Systems  Performance 
through  Design  for  Six  Sigma  (DFSS) 


1.1  The  Software  Systems  Engineering  Challenge 

Whether  we  practice  software  systems  engineering  in  the  aerospace,  commercial,  automotive,  or 
service -oriented  industries,  the  game  we  play  is  the  same:  develop  increasingly  complex  systems 
with  smaller  performance  margins  that  meet  the  users’  requirements  in  the  shortest  time,  with 
high  reliability,  while  remaining  open  and  adaptable,  and  at  the  lowest  cost.  As  software  systems 
engineers,  we  focus  on  the  basics:  define  the  requirements  and  architecture,  develop  the  plan,  de¬ 
sign  and  analyze,  and  finally  integrate,  verify,  and  validate.  We  all  know  that  these  are  the  basics 
and  you  have  to  follow  the  standard  process  or  you  can’t  even  be  in  the  game.  We  have  configura¬ 
tion  management  practices  to  assure  that  we  manage  changes  along  the  way  and  perform  contract 
modifications  and  negotiations  with  the  customer  throughout  the  lifecycle.  Our  customers  under¬ 
stand  the  process  and  adhere  to  it. 

However,  we  naturally  want  to  push  the  boundaries  of  what  is  possible  to  be  able  to  deliver  capa¬ 
bilities  to  users  in  the  least  amount  of  time  and  at  the  lowest  cost.  Companies  and  software  sys¬ 
tems  engineers  that  are  good  at  managing  their  customers  and  providing  data  to  support  contract 
specification,  schedule,  and  cost  negotiations  usually  win  the  game.  Yet  with  increasing  complex¬ 
ities,  increased  emphasis  on  mission  assurance,  aggressive  cost  reduction  targets,  shorter  time-to- 
market  requirements,  and  ever-changing  user  needs,  our  developed  systems  must  become  increas¬ 
ingly  robust  to  keep  us  ahead.  Our  processes,  tools,  training,  and  people  must  adapt  just  as 
quickly,  if  not  quicker,  than  our  competitors  to  customer  and  user  demands.  We  need  a  set  of 
proven  enablers  that  we  can  insert  into  our  existing  product  development  process  to  address  these 
challenges  in  our  software  systems  environment. 

1 .2  Design  for  Six  Sigma  (DFSS) 

In  2005,  the  Raytheon  Integrated  Defense  Systems  (IDS)  engineering  Design  for  Six  Sigma 
(DFSS)  team  was  formed  to  identify,  develop,  pilot,  and  deploy  industry  best  practice  product  op¬ 
timization  methods  and  processes  to  predict,  manage,  and  improve  performance,  producibility, 
and  affordability  for  the  benefit  of  our  customers.  DFSS  is  the  proactive  application  of  our  overall 
Raytheon  Six  Sigma  program,  focused  on  product  optimization.  DFSS  methodologies  deployed 
within  Raytheon's  Integrated  Product  Development  Process  (IPDP)  are  proven  enablers  in  the 
identification  of  critical  customer  requirements,  development  of  architectures,  optimization  of 
critical  design  parameters,  development  and  deployment  of  enabling  process  performance  models, 
analysis  of  cost  and  schedule,  and  the  successful  integration,  verification,  and  validation  of  our 
systems. 

The  Raytheon  IDS  DFSS  program  is  defined  as  having  the  following  technical  elements: 

•  voice-of-the -customer  analysis,  enabling  architecture/design  trade-off  evaluation 

•  up-front  architecture/design  trade  space  evaluation 
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•  performance  modeling,  integration,  and  analysis 

•  cost  risk  analysis  using  Monte  Carlo  simulation 

•  focused  application  of  Design  for  Manufacture  and  Assembly  (DFMA)  with  sustainment 
principles  and  practices 

•  statistical  test  optimization  (STO)  using  Design  of  Experiments  (DOE) 

•  critical  chain  project  management;  schedule  risk  assessment  and  critical  chain  for  execution 
(RAACE)  concepts  (including  that  of  schedule  risk  analysis  using  Monte  Carlo  simulation) 

•  supplier  partnering  using  DFSS  methodologies 

While  each  of  these  technical  elements  in  their  own  right  have  delivered  significant  value  to  the 
Raytheon  IDS  business,  the  focus  of  this  report  will  be  on  the  development,  piloting,  and  inte¬ 
grated  deployment  of  three  specific  core  DFSS  methodologies  that  have  most  significantly  im¬ 
pacted  our  abilities  to  predict,  manage,  and  improve  our  software  systems.  These  are  described  in 
Sections  2,  3,  and  4. 

1.3  Measures  of  Success  and  Delivered  Results 

Effective  measures  of  success  are  integral  for  determining  the  degree  of  success  in  delivering  re¬ 
sults  against  business  objectives.  The  DFSS  team  has  developed  and  deployed  an  integrated 
scorecard  using  an  Oregon  Productivity  Matrix  (OPM)  to  measure  success  against  established 
goals.  A  supporting  project  activities  database  was  also  developed  to  provide  centralized  tracking 
and  management  for  all  deployments  and  engagements  related  to  DFSS. 

Specific  measures  of  success  tracked  by  the  DFSS  OPM  against  established  annual  goals  and 
stretch  targets  include:  achieved  financial  savings,  return  on  investment  (ROI)  ratio  (financial  sav¬ 
ings/process  investment)  from  DFSS  projects,  the  number  of  DFSS  projects,  delivered  DFSS 
course  training  hours,  and  the  number  of  customer  and  supplier  DFSS  project  deployments.  DFSS 
goals  are  established  annually  in  concert  with  senior  IDS  leadership  to  enable  Raytheon  IDS  busi¬ 
ness  goals.  Established  annual  IDS  goals  are  specifically  delineated  on  not  only  the  DFSS  team 
human  resources  performance  screens,  but  also  on  the  individual  human  resource  performance 
screens  of  the  IDS  vice  president  of  engineering  and  all  functional  directors  (including  systems 
engineering,  software  engineering,  electrical  design,  and  mechanical  engineering).  The  entire 
value  stream  is  “all  in”  relative  to  the  delivery  of  DFSS  results  against  goals.  Annual  achieved 
savings  from  DFSS  projects  have  consistently  delivered  an  annual  ROI  ratio  of  over  20: 1. 

The  supporting  DFSS  project  activities  database  was  created  to 

•  track  all  DFSS  technical  element  engagements  within  IDS 

•  track  DFSS  projects  to  closure 

•  provide  a  tracking  and  document  repository  for  all  related  assets 

•  provide  multiple  users  with  access  to  track  the  progress  of  their  engagements 

•  provide  workflow  management  of  in-progress  engagements 

•  provide  historical  library  engagements  by  program,  supplier,  business,  department,  sponsor, 
POC,  and  so  forth 

•  capture  all  highlights  associated  with  monthly  progress  to  goals 
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make  it  easy  to  generate  metrics  for  leadership  by  technical  element,  achieved  cost  savings, 
business,  supplier,  year,  cost,  training,  and  so  forth 

provide  a  means  of  researching  and  sharing  lessons  learned  across  DFSS  projects 
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2  Statistical  Test  Optimization  Using  Design  of  Experiments 
and  Combinatorial  Design  Methods 


This  report  focuses  on  the  development,  piloting,  and  integrated  deployment  of  three  specific  core 
DFSS  methodologies  that  significantly  impacted  our  ability  to  predict,  manage,  and  improve  our 
software  systems.  In  this  section,  we  explain  the  first  methodology,  the  application  of  statistical 
test  optimization  (STO)  using  Design  of  Experiments  (DOE)  and  combinatorial  design  methods. 

2.1  Identification  of  Significant  Improvement  Opportunity 

As  an  industry  we  are  being  challenged  by  our  customers  and  the  marketplace  to  develop  and  de¬ 
liver  increasingly  complex  systems  with  smaller  performance  margins  that  meet  the  users’  re¬ 
quirements  in  the  shortest  time,  with  high  reliability,  an  open  and  adaptable  architecture,  and  at 
the  lowest  cost.  Given  this  challenge,  there  is  increased  pressure  on  test  activities  to  ensure  that 
software  intensive  systems  meet  all  requirements  and  expectations  given  limited  test  resources. 
Industry  studies  have  estimated  that  test  and  its  associated  rework  represent  30%  -  50%  of  the  to¬ 
tal  product  development  costs.  Given  this  investment,  test  represents  fertile  ground  for  optimiza¬ 
tion. 

Accordingly,  large-scale  efforts  are  underway  at  the  Department  of  Defense  (DoD)  to  create  a 
paradigm  shift  away  from  test  events  that  are  driven  by  combat  scenario  test  strategies  and  budget 
concerns  to  an  approach  that  is  scientific  and  statistically  rigorous.  A  guidance  document  pub¬ 
lished  by  the  DoD  director  of  operational  test  and  evaluation  provides  a  specific  request  to  “in¬ 
crease  the  use  of  both  scientific  and  statistical  methods  in  developing  rigorous,  defensible  test 
plans  in  evaluating  their  results”  [7].  This  guideline  document  requires  test  programs  to  report  evi¬ 
dence  of  well-designed  experiments,  including  continuous  response  variables,  descriptions  of  how 
test  factors  will  be  controlled  during  test,  and  the  strategy  for  exploring  the  design  space.  Similar 
statistical  test  optimization  efforts  are  being  undertaken  in  the  commercial  marketplace. 

2.2  Methodology  Development 

Traditionally,  DOE  techniques  have  been  used  to  model  and  optimize  performance  through  the 
statistical  identification  and  optimization  of  input  factors  and  interactions  that  are  statistically 
driving  performance  and  variability.  An  industry  opportunity  emerged  focused  on  extending  the 
leverage  of  DOE  within  the  test  space  with  the  motivation  of  integrating  domain  expertise  and  sta¬ 
tistical  methods  to  most  effectively  cover  the  test  space  at  the  minimum  cost  and  cycle  time. 

The  Defense  Acquisition  University  glossary  defines  DOE  as  “a  statistical  methodology  for  plan¬ 
ning,  conducting,  and  analyzing  a  test/experiment.”  DOE  allows  testers  to  provide  decision  mak¬ 
ers  with  statistically  defensible  options  for  testing  that  show  how  much  it  will  cost  to  achieve  a 
given  level  of  knowledge.  DOE  enables  a  test  planner  to  maximize  the  knowledge  gained  for  a 
given  set  of  resources  through  scientific  planning  processes.  The  purpose  of  a  designed  test  or  ex¬ 
periment  is  to  ensure  that  the  ranges  of  the  causal  variables  are  adequately  covered  to  provide  the 
most  accurate  responses  possible  for  the  fewest  number  of  experiments  and  to  answer  the  ques¬ 
tions  of  interest  while  defining  risks  to  support  with  statistically  based  decisions.  Experimental 
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design  methods  are  employed  during  test  design  planning  such  that  the  effect  of  factors  (inde¬ 
pendent  variables)  and  their  interactions  (synergistic  effects)  on  one  or  more  measured  responses 
(also  called  dependent  variables)  can  be  statistically  explored.  After  testing  is  completed  and  data 
collected,  DOE  analysis  provides  quantifiable  statistically  defensible  conclusions  about  system 
performance. 

DOE  improves  DoD  test  rigor  by  objectively  justifying  the  number  of  trials  conducted  based  on 
decision  risk,  well  apportioning  test  conditions  in  the  battle  space,  guiding  the  execution  order  to 
control  nuisance  variation,  and  objectively  separating  the  signal  of  true  system  responses  from  un¬ 
derlying  noise.  DOE  enables  effectiveness  of  system  discovery  with  detailed  process  decomposi¬ 
tion,  tying  test  objectives  to  performance  measures,  and  it  includes  test  matrices  that  span  the  op¬ 
erating  region  and  allow  for  faults  to  be  traced  to  causes.  Efficiencies  are  gained  by  combining 
highly  efficient  screening  designs  with  initial  analyses  to  learn  early,  followed  by  knowledge- 
based  test  augmentation  for  continued  learning  via  statistical  modeling,  culminating  in  validation 
tests — all  with  the  purpose  of  full  system  understanding  using  only  the  resources  necessary. 

The  menu  of  available  modern  designs  is  quite  diverse,  as  are  the  statistical  methods  of  linking 
input  changes  to  associated  output  changes  (analysis).  Since  the  experimental  units  may  be  peo¬ 
ple,  machines,  techniques,  services,  and  environmental  effects — among  others — DOE  is  a  disci¬ 
pline  that  has  applications  across  the  full  array  of  industrial,  services,  research,  hard  and  social 
sciences,  finance,  scheduling,  and  engineering.  The  discipline  was  founded  by  Sir  Ronald  Fisher, 
a  British  mathematician  and  geneticist,  in  agricultural  experimentation  in  the  1920s  and  1930s. 

His  seminal  text,  The  Design  of  Experiments,  was  published  in  1935.  Experimental  design  grew 
into  extensive  use  in  the  chemical  and  process  industries  in  the  1950s,  was  adopted  by  Japan  as 
they  re-invented  their  industrial  base  in  the  1960s,  and  saw  increased  use  in  industry  as  part  of  the 
quality  movement  in  the  1970s  and  80s.  The  modern  period  (circa  1990+),  fueled  by  increasingly 
capable  software  tools,  has  seen  worldwide  employment  of  DOE  in  all  industries  and  across  the 
product  lifecycle. 

Table  1  lists  some  of  the  most  common  experimental  designs  [6].  The  type  of  the  design  should 
always  reflect  the  goal  of  the  experiment.  Several  typical  goals  are  captured  in  the  table,  including 
characterization,  optimization,  screening,  and  testing  for  problems. 
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Table  1 :  Common  Defensible  Experimental  Designs,  from  International  Test  and  Evaluation  Associa¬ 
tion 


Test  Objectives 

Example  Design 

Method 

Examples  of  Applications 

Product  design  & 
development 

Super-Saturated  Designs 

Trade  Studies  &  Engineering  Design 
and  Analysis 

Factorial  &  Fractional 

Factorial  Designs 

Process  optimization 

Response  Surface 

Designs  (RSM) 

Trade  Studies  &  Engineering  Design 
and  Analysis.  Product  design  robust 
to  manufacturing  environ  variations 

Optimal  Designs 

Screen  for  important  factors 

Factorial  &  Fractional 

Characterizing  Performance 

Factorial  Designs 

Various,  w/  hard-to- 

randomize  factors 
(time/$$$/danger-to  change) 

Split/Strip  Plot  Designs 

Mach-Alt  Performance  -  Alt  hard 

2nd  Order  Split  Plot 

Wind  Tunnel  Optimize  -  article  cfig 

hard 

Characterize  a  system  or 
process  over  an  envelope 

Factorial  &  Fractional 

Characterizing  Performance  with 
environmental  variables;  software 
performance  testing  over  networks 

Factorial  Designs,  RSM, 
Optimal  Designs, 

Covariates 

Test  for  Problems 

Combinatorial  Designs 
(e.g.  Factor  Cover  Array 

Software  Testing  for  faults 

Orthogonal  Arrays 

Integration  &  Interoperability 

Space  Filling  Designs 

Integration  &  Interoperability 

Evaluation  of  mat'l 

properties 

Accelerated  Life  tests 

Accelerated  Life  Tests 

Bayesian  reliability 

Mixture  Designs 

Simple  Cause-Effect 

Relations 

Single  Factor  Regression 

RSM  desisns 

Material  properties,  investigations, 

exDlorations 

Item  Acceptance  Test 

One  sample  t  test 

FQT,  FCA,  LA,  surveillance,  ... 

Two  sample  t  test 

In  conjunction  with  DOE  techniques,  combinatorial  design  methods  (CDM)  are  employed  to  sta¬ 
tistically  assess  the  test  coverage  of  existing  and  under-development  test  plans.  This  is  accom¬ 
plished  by  determining  the  percentage  of  n-way  combinations  between  identified  input  test  pa¬ 
rameters  that  are  covered  by  potential  alternative  test  plans.  Specific  interest  is  given  to  those  n- 
way  combinations  that  are  of  technical  interest  from  a  domain  or  use-case  perspective.  Once  the 
key  individual  and  interoperability  requirements  have  been  identified  from  a  technical  and  use- 
case  perspective,  an  optimized  test  plan  is  developed  using  DOE  algorithms.  The  resulting  devel¬ 
oped  experimental  designs  are  mathematically  orthogonal,  thereby  enabling  root  causal  analysis. 

2.3  Piloting,  Measurement,  and  Refinement 

Effective  piloting,  measurement,  and  refinement  based  on  lessons  learned  from  integrated  pro¬ 
gram  deployment  are  essential  elements  in  process  improvement.  Therefore,  the  developed  test 
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optimization  using  DOE  methodology  and  associated  process  were  piloted  across  a  multiple  set  of 
diverse  programs  and  test  environments  to  maximize  learning  and  refinement  before  full  deploy¬ 
ment.  Expectations  were  clearly  defined  for  the  outcome  prediction:  a  quantitative  assessment  of 
existing  test  coverage  and  the  statistical  generation  of  alternative,  more  efficient  and  effective  test 
plans. 

Table  2  summarizes  the  achieved  pilot  efficiencies  that  resulted  from  statistical  test  optimization 
efforts  as  compared  with  each  of  the  individual  original  test  plans  [9].  In  each  case,  these  reduc¬ 
tions  were  achieved  while  maintaining  or  improving  upon  existing  test  coverage.  It  should  be 
noted  that  while  the  overall  number  of  test  cases  in  each  case  was  reduced,  there  were  observed 
subsections  of  testing  (most  notably  within  the  system  range  testing)  in  which  the  number  of  indi¬ 
vidual  tests  was  increased  to  achieve  the  desired  level  of  test  coverage. 


Table  2:  Achieved  Efficiencies  from  integrated  Statistical  Test  Optimization  Using  DOE  Program  Pilot¬ 
ing 


Test 

Original  Test  Plan 

Optimized  Test  Plan 

Subsystem  Scenario  Testing 

28  Tests 

8  Tests  (71%  reduction) 

Systems  Mission  Testing 

25  Missions 

18  Missions  (28%  reduction) 

Software  Simulation 

100  Runs 

40  Runs  (60%  reduction) 

System  Range  Testing 

1036  Tests 

632  tests  (39%  reduction) 

Software  Subsystem  Testing 

90  Tests 

63  Tests  (30%  reduction) 

System  Scenario  Generation 

8  Missions 

6  Missions  (25%  reduction) 

Software  Verification  Testing 

1600  Tests 

885  tests  (45%  reduction) 

System  Verification  Testing 

246  Tests 

48  tests  (80%  reduction) 

A  more  detailed  walk-through  of  the  process  used  in  achieving  these  results  is  included  as  part  of 
an  unclassified  case  study  in  Section  2.5. 

Specific  refinements  made  to  the  developed  methodology  as  a  direct  result  of  the  piloting  include 
the  following:  the  development  of  an  up-front,  three-hour  training  course  to  enable  stakeholder 
understanding  of  the  methodology;  the  development  of  two  in-parallel  complementary  processes 
(one  for  use  when  an  existing  test  plan  is  present  and  one  for  use  when  a  test  plan  is  originated); 
and  adjustments  to  the  employed  DOE  nomenclature  to  enable  quicker  alignment  with  the  termi¬ 
nology  already  in  use  by  the  various  test  teams. 

2.4  Integrated  Deployment,  Validation,  and  Sustainment 

Building  on  the  piloting  results  and  lessons  learned,  the  DFSS  team  deployed  the  developed  statis¬ 
tical  test  optimization  using  DOE  process  as  an  enabler  integrated  with  Raytheon’s  IPDP,  and 
linked  to  specific  test  planning  and  execution  process  steps.  The  integration  process  requires  de¬ 
tailed  evidence  that  the  developed  process  has  been  vetted  through  a  robust  piloting  process  or  in¬ 
dependent  review.  Because  of  this  scrutiny,  the  process  requires  a  significant  degree  of  piloting 
evidence  (note  the  piloting  of  this  process  across  eight  different  projects  and  test  environments), 
which  typically  serves  us  well.  This  case  was  no  different  with  regard  to  new  process  validation 
and  sustainment.  Results  from  full  integrated  program  deployment  have  been  validated;  an  on-av- 
erage  reduction  of  30%  was  achieved,  while  maintaining  or  improving  on  achieved  test  coverage 
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has  resulted  in  significant  business  savings  for  our  company  and  increased  customer  satisfaction. 
For  process  sustainment  purposes,  a  number  of  sustainment  assurances  have  been  put  in  place,  in¬ 
cluding  capturing  the  financial  benefits  achieved  by  individual  programs  using  STO  in  our  DFSS 
database  system,  acquiring  an  unlimited  license  agreement  for  our  primary  toolset  ( the  rdExpert 
Test  Suite,  a  COTS  tool  suite  provided  by  Phadke  Associates),  and  the  creation  of  an  appren¬ 
tice/mentor  system  for  developing,  maintaining,  and  expanding  statistical  test  optimization  using 
DOE  subject  matter  expertise. 

Raytheon  senior  leadership  and  our  customers  have  become  strong  advocates  of  this  development 
process.  A  quote  from  Raytheon  CEO  Dr.  Tom  Kennedy  reinforces  his  advocacy:  “There  is  no 
way  around  it  -  we  have  to  find  ways  to  do  more  with  less.  The  integrated  program  use  of  statisti¬ 
cal  techniques  such  as  DOE  has  proven  itself  to  be  a  powerful  enabler  in  our  test  optimization  ef¬ 
forts  to  reduce  cost  and  cycle  time  while  providing  our  customers  with  confidence  that  our  sys¬ 
tems  will  perform.” 

2.5  STO  Case  Study 

The  following  high-level,  unclassified  electronic  warfare  (EW)  subsystem  case  study  provides  a 
walk-through  of  statistical  test  optimization  using  DOE  analysis  process. 

The  list  below  describes  the  parameter  design  space  as  defined  by  the  EW  subsystem  test  team 
based  on  their  invaluable  domain  expertise: 

•  platform  type 

missile,  aircraft,  ship,  land 

•  frequency 

bandl,  band2,  band3  low,  band3  high,  band4 

•  frequency  type 

constant,  agile 

.  PRI 

CW,  very  low,  low,  medium,  high 

•  PRI  type 

CW,  constant,  switcher,  jitter,  stagger 

.  PW 

CW,  narrow,  medium,  wide 

•  Scan 

none,  fast,  medium,  slow 

•  Scan  type 

steady,  circular,  conical,  sector 

Note  that  the  parameter  design  space  is  defined  based  on  an  understanding  of  the  subsystem  re¬ 
quirements  and  operational  needs. 

Next  we  will  define  test  constraints  (test  parameter  combinations  that  are  infeasible). 
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CW-related  constraints 


•  PRI  [CW]  with  PRI  type  [constant,  switcher,  jitter,  or  stagger] 

•  PRI  type  [CW]  with  PRI  [very  low,  low,  medium,  or  high] 

•  PW  [CW]  with  PRI  type  [constant,  switcher,  jitter,  or  stagger] 

•  PW  [CW]  with  PRI  [very  low,  low,  medium,  or  high] 

Scan-related  constraints 

•  scan  [none]  with  scan  type  [circular,  conical,  or  sector] 

•  scan  type  [steady]  with  scan  [fast,  medium,  or  slow] 

Assessing  the  existing  plan  using  combinatorial  design  methods  (note  that  the  original  plan 
developed  using  domain  expertise  solely  without  the  use  of  DOE): 

Original  test  plan 

•  90  test  cases 

Existing  coverage  analysis  results: 

•  critical  domain  (mains  and  2-way)  coverage:  84% 

•  overall  domain  (mains  and  2-,  3-,  4-way)  coverage:  54.6% 

•  2-way  combinations:  67.9% 

•  3-way  combinations:  35.2% 

•  4-way  combinations:  15.5% 

•  missing  2-way  combinations:  144  (168  Total  -  24  constraints) 
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Figure  1:  Existing  Test  Coverage  Analysis  [9] 


Optimized  alternative  test  plan  using  statistical  test  optimization  using  DOE: 


Optimized  test  plan 

•  49  test  cases  proposed  (45%  reduction) 

parameter  combinations  that  yield  the  highest  2-way  combinations  coverage  of  the 
test/capability  space,  that  is,  capability-based  (tests  each  parameter  class  with  every 
other  parameter  class  at  least  once) 
improved  3-  and  4-way  combinations  coverage  also 

opportunity  for  further  reduction  in  number  of  test  cases  through  exclusion  of  parameter 
pairs  of  lesser  value/interest  than  others  (further  analysis/consultation  required) 

Coverage  analysis  results 

•  critical  coverage:  100%  (16%  improvement) 

•  overall  coverage:  71.1%  (16%  improvement) 

•  2-way  combinations:  100%  (32%  improvement) 

•  3-way  combinations:  61.3%  (26%  improvement) 

•  4-way  combinations:  23%  (7%  improvement) 

•  missing  2-way  combinations:  0 
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Test  No.  4  8  12  16  20  24  28  32  36  40  44  48 


Critical  DC 
Ortirall  OC 


53181  70  7?1  78153  84  317  87  613  92230  9193?  96  817  98  086  98  986  99  66?  99  887 

?3  815  39  715  45  510  50  597  53  907  58  1?6  61  196  63  691  65  759  67  503  69  325  70  732 


Critical  DC 


Overall  DC 


Figure  2:  Optimized  Test  Coverage  Analysis  [9] 


Not  only  is  the  optimized  plan  superior  from  an  efficiency  perspective  (49  tests  versus  90  tests), 
but  also  from  an  n-way  coverage  perspective.  Reinforcing  the  importance  of  increased  n-way  cov¬ 
erage,  an  industry-wide  NIST  analysis  study  across  a  large  number  of  software  STO  deployments 
indicates  a  strong  statistical  correlation  between  improved  defect  containment  and  increased  n- 
way  test  coverage  [10]. 


2.6  Sharing  of  Best  Practices  and  Lessons  Learned 

Collaborative  sharing  across  individual  programs  within  Raytheon  IDS,  Enterprise  Raytheon,  and 
the  industry  is  integral  to  our  achieved  results  to  date  and  our  drive  for  continuous  improvement. 
Below  is  an  affinitized  list  of  our  key  lessons  learned  for  achieving  statistical  test  optimization 
high  maturity. 

Organizational  readiness 

•  STO  is  a  core  test  planning  competency. 

•  There  are  development  processes  for  STO  practitioners  and  subject  matter  experts  (SMEs). 

•  SMEs  have  an  external  bias;  they  are  always  searching  for  best  practices. 

•  Trainers  and  coaches  are  experienced  and  motivational. 

•  STO  experts  and  practitioners  are  recognized  for  their  results. 

•  The  SME  network  is  well  led/networked. 

•  STO  is  actively  promoted  and  expected  by  leadership. 

•  Success  stories  are  propagated  throughout  the  organization. 
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•  Success  is  celebrated  (“OctoberTest”). 

•  The  organization  works  to  overcome  traditional  “one  factor/requirement  at  a  time”  testing 
mentality. 

Process  and  tools 

•  Use  of  scientific  test  and  analysis  techniques  is  integrated  directly  into  the  product  develop¬ 
ment  process,  procedure,  and  reviews  as  a  part  of  standard  design  practice. 

•  Test  readiness  reviews  focus  on  evaluating  developed  test  designs,  not  on  whether  or  not 
DOE  was  used. 

•  STO  efforts  are  integrated  rather  than  run  in  parallel  with  traditional  methods. 

•  Recognition  exists  that  process  experts  tend  to  underestimate  context;  domain  experts  tend  to 
overrate  context;  the  truth  is  in  the  middle. 

•  Test  is  automated  through  scripting. 

•  Be  wary  of  the  tendency  for  statistical  tool  infatuation  (it’s  better  to  be  tool  agnostic). 

•  Opportunities  for  up-front  integrated  operational  analysis,  model-based  engineering,  and  Ag¬ 
ile  application  are  leveraged. 

Program  integration 

•  STO  is  integral  to  up-front  program  planning  (“festina  lente”). 

•  A  multi-discipline  test  optimization  workshop  is  provided  upfront. 

•  Implementation  is  driven  by  product  teams,  not  by  SMEs. 

•  Analysis  output  is  linked  to  the  risk  and  opportunity  register. 

•  STO  is  integral  to  the  test  readiness  review  process. 

•  Up-front  measurement  system  analysis  is  in  place. 

•  Integrated  and  contextual  understanding  exists  of  measures  of  effectiveness  (statistical  confi¬ 
dence,  coverage,  and  power). 

•  Be  wary  of  statistical  tampering  of  individual  test  cases  and  experimental  runs  by  test  leads 
and  operators. 

•  Test/DOE  language  issues  (test  cases,  presentations,  objectives,  missions,  scenarios,  factors, 
parameters,  etc.)  can  slow  the  process  and  the  technical  context  understanding. 


CMU/SEI-2017-TR-001  |  SOFTWARE  ENGINEERING  INSTITUTE  |  CARNEGIE  MELLON  UNIVERSITY 
[DISTRIBUTION  STATEMENT  A]  This  material  has  been  approved  for  public  release  and  unlimited  distribution. 


12 


3  Process  Performance  Modeling  and  Analysis 


This  section  focuses  on  the  second  of  three  core  DFSS  methodologies  we  used  to  improve  our 
ability  to  predict  and  manage  our  software  systems:  process  performance  modeling  and  analysis. 

3.1  Identification  of  Significant  Improvement  Opportunity 

The  Raytheon  IDS  business  and  the  DFSS  team  were  first  exposed  to  process  performance  mod¬ 
els  through  the  SEI-led  development  of  the  CMMI  model.  Process  performance  baselines  and  pro¬ 
cess  performance  models  are  expected  CMMI  high  maturity  (maturity  levels  4  and  5)  artifacts  that 
build  on  the  measurement  and  analysis  activities  established  at  CMMI  level  2  to  lift  an  organiza¬ 
tion  from  a  reactive  management  state  to  a  proactive  management  state.  By  aligning  process  per¬ 
formance  baselines  and  business  objectives,  process  performance  baselines  and  models  act  as  fa¬ 
cilitators  to  organizational  and  project  success. 

Because  of  their  analytical  expertise,  the  DFSS  team  serves  the  IDS  engineering  organization  as 
statistical  modeling  and  analysis  SMEs  in  the  development,  piloting,  deployment,  validation,  and 
sustainment  of  process  performance  models  that  align  with  and  enable  business  success.  A  suite 
of  developed  process  performance  models  (PPMs)  have  been  effectively  developed  and  deployed 
by  Raytheon  IDS.  Each  PPM  is  identified  and  selected  based  on  its  individual  business  return  to 
the  Raytheon  IDS  business.  The  direct  relationship  between  business  objectives  and  quality  and 
process  performance  objectives,  and  the  use  of  process  performance  baselines  and  models  to 
quantitatively  manage  our  progress  toward  achieving  these  objectives,  focused  our  organization 
on  the  characteristics  of  success.  If  organizational  objectives  are  subjective,  the  project  may  be¬ 
come  disengaged  with  those  objectives.  Making  process  performance  baselines  and  quality  and 
process  performance  objectives  an  integral  part  of  project  management  clarifies  each  project’s 
role  in  business  success. 

The  versatility  of  the  goal  question  metric  (GQM)  approach  further  enabled  our  development  and 
deployment  of  PPMs  in  the  form  of  a  goal  question  model  approach.  As  with  process  performance 
baselines,  all  PPM  efforts  are  initiated  from  the  linkage  and  alignment  of  quality  and  process  per¬ 
formance  objectives  and  business  objectives.  This  approach  is  absolutely  critical  to  effective  pro¬ 
cess  performance  modeling  since  without  this  up-front  business  alignment,  there  is  a  tendency  to 
create  elegant  models  rather  than  effective  models  that  support  business  objectives. 

In  this  report,  the  Systems  Lifecycle  Analysis  Model  (SLAM)  PPM,  developed  and  deployed  by 
Raytheon  IDS,  is  discussed  as  a  representative  software  systems  application  case  study  in  order  to 
provide  an  explanation  of  the  employed  methodology  and  share  the  lessons  learned. 

3.2  Methodology  Development 

Development  of  our  SLAM  process  performance  model  was  a  direct  result  of  a  business  concern 
expressed  by  Raytheon  IDS  leadership  about  the  productivity  and  rework  risks  associated  with  ac¬ 
celerated  concurrent  engineering  efforts  and  their  potential  downstream  impact  on  downstream 
cost  performance.  This  concern  naturally  led  to  our  generation  of  questions  around  what  factors 
related  to  our  concurrent  engineering  efforts  influence  our  achievement  of  cost  performance,  and 
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the  controllable  sub-processes  related  to  those  factors.  Of  specific  interest  in  the  development  of 
the  SLAM  model  was  the  potential  statistical  relationship  between  requirements  volatility  and  the 
degree  of  requirements/design  overlap  with  that  of  downstream  software  development  cost  perfor¬ 
mance.  Modeling  the  relationship  between  potential  input  factors  and  project  outcomes  involved 
the  use  of  statistical  methods  such  as  regression  analysis  and  Monte  Carlo  simulation. 

The  resulting  SLAM  model  enables  projects  with  aggressive  schedules  in  which  software  design 
activities  are  planned  to  begin  prior  to  requirements  release  are  able  to  quantify  the  associated  cost 
performance  risk,  determine  the  requirements  volatility  level  that  must  be  maintained  to  meet  cost 
performance  objectives,  and  identify  the  process  changes  necessary  to  manage  requirements  ac¬ 
cordingly. 

SLAM  Model  output  and  things  key  stakeholder  care  about  include  the  following: 

•  outcome  prediction:  confidence  interval  estimation  of  cost  performance  (generated  using 
Monte  Carlo  simulation)  utilizing  a  developed  multi-factor  regression  model 

•  for  key  stakeholders,  the  outcome  prediction  is  of  critical  importance  to  them  for  these  rea¬ 
sons: 

systems/software  engineering:  enables  integrated  engineering  team  risk  assessment  and 
sensitivity  analysis  around  the  likelihood  of  achieving  cost  performance  objectives  and 
the  development/deployment  of  mitigation  strategies. 

quality  engineering:  reinforces  the  importance  of  the  development  and  deployment  of 
up-front  project  quality  planning  and  analysis,  paving  the  way  for  their  value-added  in¬ 
volvement 

Factors  used  in  the  SLAM  process  performance  model  include  the  following: 

•  requirements  volatility 

post  formal  requirements  document  release  change  % 

requirements  volatility  is  a  required  measurement  collected  and  reported  by  every  devel¬ 
opment  project  across  Raytheon  Company 

organizational/project  collected  baselines  were  stratified  by  product  type 
for  simulation  input  purposes,  a  Gaussian  approximation  is  typically  employed 

•  requirements/design  lifecycle  overlap 

percent  of  the  software/hardware  design  budget  expended  (as  measured  by  our  earned 
value  management  system)  at  the  time  of  formal  requirements  document  release 
non-standard  project  measurement  collected  and  analyzed  during  the  SLAM  develop¬ 
ment  piloting  and  deployment 

for  simulation  input  purposes,  a  Gaussian  approximation  is  typically  employed 

In  the  specific  case  of  the  SLAM  model,  a  mathematical  function  of  the  input  factors  was  reasona¬ 
bly  well  correlated  with  the  output  responses  using  linear  regression  techniques  (with  an  adjusted 
r-squared  value  =  0.65,  p  =  0.00).  See  Figure  3  for  the  graphical  portrayal  of  the  data.  The  regres¬ 
sion  equation  associated  with  this  statistical  correlation  was  the  building  block  for  our  SLAM 
model  development  efforts. 
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Healthy  ingredients  of  process  performance  models  as  defined  by  the  SEI  include  the  following: 

•  are  statistical,  probabilistic,  or  simulation  in  nature 

•  predict  interim  and/or  final  project  outcomes 

•  use  controllable  factors  tied  to  sub-processes  to  conduct  the  prediction 

•  model  the  variation  of  factors  and  understand  the  prediction  range  or  variation  of  the  out¬ 
comes 

•  enable  “what-if  ’  analysis  for  project  planning,  dynamic  re-planning,  and  problem  resolution 
during  project  execution 

•  connect  “upstream”  activity  with  “downstream”  activity 

•  enable  project  to  achieve  midcourse  corrections  to  ensure  project  success 

These  ingredients  have  served  us  well  in  guiding  our  process  performance  model  development 
and  deployment  efforts.  The  SLAM  model  leverages  the  statistical  correlation  of  controllable  fac¬ 
tors  to  an  outcome  prediction  in  the  form  of  a  regression  equation.  A  user-friendly  Excel-based 
interface  was  created  using  industry  available  COTS  tools  (either  Crystal  Ball  or  @Risk  -  we 
have  provided  an  either/or  option  to  enable  users  with  different  licensing  constraints)  to  model 
and  statistically  generate  a  cost  performance  prediction  interval  using  Monte  Carlo  simulation 
(see  Figure  4). 
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Figure  4:  SLAM  PPM  Model  User  Interface  [1 1  ] 

Note  that  the  SLAM  model  user  interface  also  includes  worksheets  containing  Crystal  Ball/@Risk 
download  instructions,  step-by-step  guidance  for  projects  on  “Running  SLAM”  and  an  “Interpret¬ 
ing  the  Results”  guide. 

3.3  Piloting,  Measurement,  and  Refinement 

SLAM  pilot  data  collection: 

•  Requirements  volatility 

a  required  measurement  collected  and  reported  by  every  development  project  across 
Raytheon  Company 

Organizational  innovation  and  deployment  (OID)  considerations  led  to  initiating  the  col¬ 
lection  of  requirements  volatility  data  at  the  configuration  item  (Cl)  or  module  level  to 
support  SLAM  piloting. 

•  Requirements  /  design  lifecycle  overlap 

non-standard  project  measurement  collected  and  analyzed  during  the  SLAM  develop¬ 
ment  piloting  and  deployment 

OID  considerations  led  to  the  SLAM  piloting  effort  working  closely  with  a  cross-section 
sampling  of  our  IDS  development  projects  in  defining  an  objective  measurement  that  is 
easily  collected  and  readily  available. 
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Up-front  collection-defining  dialogue  with  OID  SLAM  pilot  project  teams  provided 
highly  valuable  analytical  and  deployment  insight. 

Specific  key  takeaways  from  SLAM  piloting 

•  All  stakeholder  groups  and  projects  found  the  SLAM  model  easy  to  use  and  conceptually 
aligned  with  project  issues. 

•  The  majority  of  projects  identified  and  implemented  specific  improvements  to  the  projects’ 
defined  process  as  a  direct  result  of  SLAM  modeling  and  analysis  including: 

improving  the  systems  requirements  review  process  and  execution 

increasing  the  degree  of  concurrent  engineering  between  systems  and  software/hardware 
engineering 

increased  use  of  critical  chain  concepts  in  order  to  shorten  design  cycle  time 

•  All  projects  used  SLAM  in  performing  risk  assessments. 

•  Collected  project  data  from  SLAM  piloting  further  confirmed  the  strength  of  this  underlying 
relationship  as  defined  by  the  developed  regression  equation  model. 

3.4  Integrated  Deployment,  Validation,  and  Sustainment 

Effective  development  and  deployment  of  process  performance  baselines  and  models  have  signifi¬ 
cantly  improved  upon  our  process  alignment  and  performance  against  Raytheon  IDS  business  and 
engineering  objectives.  Resulting  efforts  in  the  areas  of  statistical  process  management,  root  cause 
analysis  and  corrective  action,  interdependent  execution,  and  statistically-based  risk  assessment 
have  resulted  in  increased  productivity,  reduced  rework  and  improved  cost  and  schedule  perfor¬ 
mance.  ROI  analysis  pertaining  to  our  Raytheon  IDS  CMMI  high  maturity  efforts  has  indicated  a 
24:1  return  from  our  process  investment.  Process  performance  baselines  and  models  are  the  spark 
that  ignited  these  results  and  fuels  our  drive  for  more. 

Specific  key  takeaways  from  SLAM  deployment,  validation,  and  sustainment 

•  Engineering  process  group  (EPG)  process  funding  was  invested  in  the  development  and  de¬ 
ployment  of  the  SLAM  Model  with  an  estimated  improvement  of  5.5%  -  1 1.5%  for  software 
development  CPI  (dependent  on  established  requirements  volatility  baseline). 

•  Based  on  its  deployment  results,  SLAM  has  been  formally  integrated  into  our  development 
process;  accordingly,  SLAM  was  formally  released  to  the  Raytheon  Process  Assets  Library 
(RPAL). 

•  In  addition  to  deployed  results,  SLAM  data  was  cited  by  the  IDS  Software  Engineering  Di¬ 
rector  during  schedule  negotiations  with  program  management. 

•  SLAM  usage  is  documented  in  the  IDS  Engineering  Standards,  Engineering  Measurement 
and  Analysis  Plan  and  supporting  templates. 

•  Additional  process  funding  is  set  aside  for  maintaining  and  updating  the  SLAM  tool  and  sup¬ 
porting  its  continued  project  deployment. 

•  With  one  exception  (believed  to  have  data  integrity  issues),  all  projects  found  model  predic¬ 
tions  to  be  aligned  with  project  actuals. 
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3.5  Sharing  of  Best  Practices  and  Lessons  Learned 


Integral  to  the  process  performance  modeling  deployment  process  is  the  collection  of  specific  (to 
the  individual  model,  in  this  case  SLAM)  and  general  best  practices  and  lessons  learned. 

Specific  SLAM  lessons  learned  (in  terms  of  mitigation  strategies): 

SLAM  lessons  learned  for  reducing  requirements  volatility  and  its  effects  on  project  performance 

•  Improve  requirements  reviews. 

•  Increase  the  degree  of  concurrent  engineering  between  systems  and  hardware/software. 

•  Increase  understanding  of  the  customer  value  equation. 

•  Add  more  experienced  personnel  to  the  team. 

•  Improve  team  communications. 

•  Provide  for  increased  coaching  and  mentoring. 

SLAM  lessons  learned  for  reducing  the  degree  of  overlap 

•  Create  increased  awareness  relative  to  the  involved  risks 

•  Release  earlier,  followed  by  successive  implementation  iterations. 

•  Delay  detailed  design  efforts. 

•  Use  techniques  such  as  critical  chain  to  shorten  development  cycle  time  by  reducing  the 
waste  and  running  in  parallel  where  non-critical. 

•  Release  even  later,  but  with  exceptional  quality  (low  volatility)  in  order  to  reduce  churn  and 
rework  driving  up  cost. 

General  PPM  best  practices  and  lessons  learned  (in  terms  of  challenges)  include  the  following: 

•  identifying  the  correct  x  factors  to  model 

•  obtaining  sufficient  and  meaningful  data 

•  verifying  data  quality  and  integrity 

•  initial  project  team  engagement 

All  engineering  disciplines  need  to  be  present  with  their  project  plans/data  at  the  deploy¬ 
ment  meeting  in  order  to  produce  the  best  results  (for  effective  dialogue  and  brainstorm¬ 
ing  and  recognition  of  interdependence). 

•  internal  Crystal  Ball  and  @Risk  download  issues 

•  Some  users  entered  in  data  beyond  the  range  used  to  calibrate  the  model.  Model  was  updated 
to  caution  users. 

•  documenting  evidence  of  use  of  models 

The  effective  development  and  deployment  of  the  process  PPMs  has  sparked  increased  Raytheon 
IDS  interest  in  the  following: 

•  interdependent,  integrated  business  execution 

•  statistically-based  project  performance  risk  assessment 

•  identification  of  leading  indicators  that  statistically  drive  project  performance 
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statistical  modeling  and  analysis  supporting  tools  and  training 
follow-on  statistical  modeling  and  analysis  efforts 

business  investment  in  process  performance  modeling  throughout  the  product  development 
lifecycle 
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4  Cost  and  Schedule  Risk  Analysis  Using  Monte  Carlo 
Simulation 


In  this  section,  we  explain  our  use  of  cost  and  schedule  risk  analysis  using  Monte  Carlo  simula¬ 
tion,  the  final  of  the  three  core  DFSS  methodologies  described  in  this  report. 

4.1  Identification  of  Significant  Improvement  Opportunity 

Given  increased  industry  competition  and  heightened  customer  expectations  in  the  form  of  lower 
costs  and  shortened  delivery  cycle  time,  it  is  more  critical  than  ever  that  we  understand  the  uncer¬ 
tainty  associated  with  cost  estimation  and  scheduling  to  ensure  successful  execution  and  proactive 
risk  and  opportunity  management. 

As  stated  in  a  recent  SEI  technical  report,  “Difficulties  with  accurately  estimating  the  costs  of  de¬ 
veloping  new  systems  have  been  well  documented,  and  cost  overruns  in  new  systems  develop¬ 
ment  are  well  known.  The  headline  of  a  recent  defense  magazine  article  referred  to  the  true  cost  of 
a  weapon  as  ‘anyone’s  guess,’  reflecting  this  widely  acknowledged  fact.”[4]  Not  to  be  outdone  by 
the  challenges  in  cost  estimation,  there  may  be  no  more  daunting  challenge  than  that  of  schedule 
pressure.  Countless  jobs  are  falling  under  the  category  of  “Yes  we  can  do  it,  but  not  with  that 
schedule!” 

A  number  of  qualitative  factors  further  exacerbate  the  industry  cost  estimation  and  schedule  plan¬ 
ning  problems,  including  the  following: 

•  need  for  cost  estimates  driven  earlier  and  earlier  in  the  acquisition  lifecycle  before  a  tech¬ 
nical  baseline  is  established 

•  increasing,  competitive  pressure  to  be  aggressive  enabling  some  individual  task  activity  esti¬ 
mators  to  become  overly  optimistic  in  their  assumptions  (half-full  perspective) 

•  conservative  estimation  by  some  individual  task  activity  estimators  based  on  their  fear  that 
management  will  make  some  cuts  (half-empty  perspective) 

•  without  estimated  ranges  around  each  of  the  individual  task  activity  estimates,  it  is  difficult 
for  management  or  reviewers  to  identify  which  estimators  are  being  aggressive  and  which 
are  being  conservative  without  playing  a  game  of  20  questions  and  conducting  a  thorough 
examination  of  the  supporting  historical  data  set 

Use  of  cost  risk  analysis  (CRA)  and  schedule  risk  analysis  (SRA)  with  Monte  Carlo  simulation 
complements  and  greatly  improves  upon  traditional  deterministic  methods  in  quantitatively  as¬ 
sessing  the  risk  and  opportunity  associated  with  cost  estimation  and  schedule  planning,  enabling 
increased  sensitivity,  trade  space  analysis,  and  implementation  of  strategies  for  cost  risk  mitiga¬ 
tion  and  opportunity  capture. 

4.2  Methodology  Development 

Monte  Carlo  simulation  is  used  to  perform  risk  analysis  by  building  a  mathematical  model  based 
on  the  substitution  of  a  range  of  values  (in  the  form  of  a  probability  distribution)  instead  of  a  point 
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estimate  for  any  parameter  that  has  inherent  uncertainty.  In  short,  Monte  Carlo  simulation  recog¬ 
nizes  that  there  is  uncertainty  around  each  of  our  estimated  parameters  and  takes  it  into  account. 
Historical  actuals  as  well  as  information  relative  to  the  context  being  estimated  (similar  to  product 
actuals,  product  reuse,  etc.)  are  used  to  estimate  the  range  or  distribution  for  cost  for  each  work 
product.  The  rationale  is  recorded  for  each  estimated  range  to  ensure  that  we  are  objectively  cap¬ 
turing  the  basis  behind  each  of  these  estimated  ranges.  Typically,  because  of  its  ease  of  use  and 
flexibility,  the  triangular  distribution  is  used  with  three  inputs:  low  (5th  and  1st  percentile),  most 
likely,  and  high  (95th  and  99th  percentile).  The  triangular  distribution  is  highly  flexible  because  it 
can  be  easily  skewed  to  mimic  the  form  of  almost  any  distribution.  Once  each  input  parameter  has 
been  estimated,  a  histogram  is  generated  of  the  statistical  probability  distribution  of  predicted  total 
cost/effort  for  selected  confidence  levels  through  the  statistical  random  drawing  (typically  of  the 
order  of  1,000  SRA  and  10,000  CRA  data  points)  from  each  of  the  underlying  sub-product  distri¬ 
butions.  Monte  Carlo  simulation  output  analysis  also  provides  valuable  practical  insight  into  the 
key  drivers  of  cost  and  schedule  variability  and  enable  sensitivity  analysis. 

Cost  and  schedule  risk  analysis  using  Monte  Carlo  simulation  has  the  following  benefits: 

•  leverages  existing  historical  actuals,  similar-to  data,  engineering  insight,  and  Monte  Carlo 
simulation 

•  enables  projects  to  statistically  estimate  cost  and  schedule,  quantify  the  risk  and  opportunity 
associated  with  meeting  cost  and  schedule  targets,  identify  cost  and  schedule  drivers,  and 
perform  sensitivity  analysis 

•  enables  non-binding  budgetary  estimate  (NBBE)  and  schedule  “what  if’  scenario  analyses 
Inputs  to  the  simulation  include  the  following: 

•  fully-networked  schedule  (integrated  master  schedule)  which  has  been  validated  as  a  robust, 
predictive  model 

•  cost  estimates  for  tasks  in  the  format  of  three-point  estimates 

•  schedule  durations  estimated  for  select  tasks  and  high-risk  areas  using  three-point  estimates 

•  probability  distributions  assigned  (the  triangular  distribution  is  typically  employed  because 
of  its  robustness) 

•  rationale  required  for  selection  of  distribution  parameters 
Outputs  include  the  following: 

•  probability  distributions  (histograms)  of  predicted  overall  cost,  overall  schedule,  and  specific 
schedule  milestones 

•  predicted  overall  cost  and  schedule  for  selected  statistical  confidence  levels 

•  identification  of  statistical  drivers  enabling  sensitivity/risk  analysis 
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Figure  5:  Example  Cost  Risk  Analysis  (CRA)  Output 


Inputs 


IMS 


IMS  Model  Validated 

l 


Outputs 

Histogram:  Probabilities  of  Meeting  Key  Milestones 
Tornado:  Identification  of  Schedule  Drivers 


SRA  Report 


Program  Specific  Analysis  of 
results  &  recommendations 


Figure  6:  Example  Schedule  Risk  Analysis  (SRA)  Output 
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4.3  Piloting,  Measurement,  and  Refinement 


Piloting  a  new  CRA  model  and  a  revamped  SRA  execution  model  was  instrumental  in  the  matura¬ 
tion  of  these  processes.  Piloting  demonstrated  the  value  and  helped  to  get  buy-in  from  leadership 
and  end-users  before  more  widespread  deployment. 

Key  piloting  takeaways: 

•  Stakeholder  groups  and  projects  found  CRA  and  SRA  models  both  easy  to  use  and  conceptu¬ 
ally  aligned  with  project  issues. 

•  Providing  subject  matter  experts  (SMEs)  to  facilitate  the  CRA  and  SRA  processes  with 
teams  aided  adoption  and  change  management 

•  Projects  identified  and  implemented  specific  improvements  to  their  processes  and  their  exe¬ 
cution  as  a  direct  result  of  CRA  and  SRA  piloting  that  enabled  cost  and  cycle  time  reduc¬ 
tions  and  proactive  risk  and  opportunity  management,  including: 

process  redesign  based  on  critical  path  dependency 
resource  reallocation  and  conflict  resolution 

increased  investment  up  front  in  planning,  analysis,  and  training  activities  in  order  to  en¬ 
able  execution  speed  (“festina  lente”) 
minimized  churn  through  enhanced  peer  review 

provided  increased  quantitative  understanding  of  overall  cost  and  schedule  estimation 
risk  and  opportunity  and  associated  statistical  drivers  enabling  prioritization  and 
cost/risk-benefit  evaluation  of  cost/schedule  optimization  action  plans 
in  addition  to  deployed  projects,  SRA  has  been  used  up-front  during  the  bid  and  proposal 
phase  and  is  used  by  leadership  to  develop  negotiation  strategies  and  shape  contractual 
terms  and  conditions 

4.4  Integrated  Deployment,  Validation,  and  Sustainment 

After  successful  pilots  and  organic,  bottom-up  growth,  CRAs  and  SRAs  became  integrated  into 
standard  business  practices  through  senior  leadership  buy-in  and  top-down  requirement  modifica¬ 
tions.  SRAs  are  required  for  all  programs  with  the  exception  of  non-schedule  driven  efforts,  and 
CRAs  are  required  for  development  proposal  engineering  reviews.  Systems  Architecture  Design 
and  Integration  Directorate  (SADID)  functional  reviews,  and  System  Validation  Test  and  Analy¬ 
sis  Directorate  (SVTAD)  functional  reviews,  and  they  are  recommended  for  other  functional  ar¬ 
eas. 

Key  transition-to-sustainment  takeaways 

•  Finance  process  funding  was  invested  in  the  development  and  deployment  of  the  revamped 
schedule  risk  analysis  model. 

•  Engineering  process  funding  was  invested  in  the  development  and  deployment  of  the  new 
cost  risk  analysis  model. 

•  The  deployment  of  cost  risk  analysis  has  resulted  in  increased  proposal  efficiency  and  speed 
in  development  of  proposals. 
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•  The  deployment  of  schedule  risk  analysis  has  resulted  in  an  objective  understanding  of 
schedule  risks,  opportunities,  and  sensitivity  drivers,  thus  enabling  data-driven  decision¬ 
making  and  action  plans. 

•  Coupled  with  a  price-to-win  confidence  interval  developed  in  parallel,  CRA  output  has  ena¬ 
bled  Raytheon  IDS  leadership  to  make  decisions  about  competitive  pricing  alternatives  and 
their  associated  risk  and  opportunity. 

•  CRAs  have  been  formally  and  fully  integrated  into  our  product  development  and  engineering 
review  processes. 

•  SRAs  have  been  formally  and  fully  integrated  into  leadership  program  reviews  and  program 
management  plans. 

•  Additional  process  funding  is  set  aside  for  maintaining  and  updating  the  supporting  CRA  and 
SRA  tool  sets  and  supporting  continued  project  deployment. 

4.5  Sharing  Best  Practices  and  Lessons  Learned 

As  CRAs  and  SRAs  matured,  it  was  essential  to  continuously  improve  and  leverage  best  practices 
and  lessons  learned  and  facilitate  knowledge  sharing  across  the  business.  The  CRA  and  SRA  tech¬ 
nical  element  owners  leveraged  their  functional  organizations  (engineering  and  finance,  respec¬ 
tively),  the  DFSS  organization,  and  other  communication  and  collaboration  forums  such  as  com¬ 
munities  of  practice,  business-level  and  enterprise-level  councils,  and  other  partnerships —  both 
internal  and  external  to  Raytheon. 

Key  lessons  learned 

•  Alignment  across  all  levels  of  the  business  on  the  vision,  skills,  incentives,  resources,  and 
action  plans  is  required  to  roll  out  new  processes. 

•  Deployment  of  new  processes  via  a  centralized  SME  model  drives  consistency  and  quality, 
accelerates  learning,  and  fosters  change  management. 

•  SMEs  provide  an  objective,  independent  perspective  unaffected  by  project  politics  or  culture 
and  highlight  risks  and  opportunities  that  are  easily  overlooked  by  those  too  close  to  the  initi¬ 
ative. 

•  A  focus  on  business  value  and  providing  objective,  data-driven,  actionable  recommendations 
and  insights  is  necessary. 

•  Use  readily  available  commercial-off-the-shelf  (COTS)  simulation  tools,  such  as  @Risk, 
Crystal  Ball,  and  Primavera  Risk  Analysis. 
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5  Summary:  Enabling  Integrated  Project  Team  Performance 
Using  Design  for  Six  Sigma 


5.1  Bringing  It  All  Together  in  an  Integrated  Project  Plan 

The  Raytheon  IDS  engineering  DFSS  team  was  formed  to  identify,  develop,  pilot,  and  deploy  in¬ 
dustry  best  practice  product  optimization  methods  and  processes  to  predict,  manage,  and  improve 
performance,  producibility,  and  affordability  for  the  benefit  of  our  customers.  Accordingly,  the 
effectiveness  of  the  IDS  DFSS  program  is  largely  dependent  on  our  ability  to  enable  individual 
project  team  performance. 

Integral  to  the  product  development  effort  is  the  Integrated  Product  Development  System  (IPDS) 
developed  by  Raytheon  Enterprise.  IPDS  is  a  system  of  enterprise-common  processes,  process- 
related  assets  and  enablers,  reference  and  training  materials,  deployment  materials,  and  support 
services  that  enable  repeatable  and  efficient  program  capture,  planning,  and  execution.  At  the 
heart  of  this  system  is  the  IPDP,  which  is  a  collection  of  common,  tailorable,  multi-discipline  pro¬ 
cesses  applied  across  all  businesses  describing  what  information  must  be  captured  to  execute 
product  development,  production,  and  support  programs.  As  stated  by  Mark  Russell,  Raytheon 
Vice  President  of  Engineering,  Technology,  and  Mission  Assurance,  "The  Integrated  Product  De¬ 
velopment  Process  eliminates  doubt  and  provides  more  predictable  program  results,  therefore  al¬ 
lowing  our  employees  to  focus  their  creativity  where  it  belongs — on  innovating  solutions  for  our 
customers." 

As  one  would  imagine,  approval  for  the  inclusion  of  a  developed  process  and  enablers  in  IPDS 
requires  rigorous  proof  that  the  development  methodology  is  technically  sound  in  principle  and 
has  been  effectively  piloted  and  delivered  performance  results  to  project  teams.  Once  a  developed 
DFSS  process  and  its  set  of  supporting  enablers  has  been  successfully  piloted  they  are  included  in 
IPDS,  moving  them  from  best  practice  to  standard  practice.  And  since  the  DFSS  team  is  always  in 
the  process  of  identifying,  developing,  piloting,  and  deploying  best  practices  to  enable  our  pro¬ 
jects,  there  is,  in  effect,  always  a  pipeline  of  DFSS  processes  and  methods  that  we  are  looking  to 
move  from  best  practice  to  standard  practice.  Accordingly  at  program  kickoff,  each  integrated 
project  team  builds  a  “go  forward”  project  plan  using  IPDS  and  tailoring  it  to  fit  their  context.  As 
with  any  other  element  of  IPDS,  DFSS-developed  standard  processes/methods  are  included  in  this 
tailoring. 

5.2  In-process  Validation  of  Achieved  Execution  Results  Against  Plan 
and  Refinement 

The  DFSS  team  supports  this  integrated  project  planning  effort,  providing  key  subject  matter  ex¬ 
pertise,  training,  and  tools  as  needed  to  enable  the  DFSS  integration  into  the  project  plan  and  for 
defining  expectations  and  delivering  results  against  that  expectation.  In  order  to  assure  that  the  ex¬ 
ecution  by  the  integrated  project  team  aligns  with  the  developed  project  plan,  independent  reviews 
are  integral  to  all  project  lifecycle  gate  reviews.  The  results  achieved  from  the  integrated  DFSS 
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program  deployments  are  then  analyzed  against  expectations  and  used  to  update  historical  base¬ 
lines  and  prediction  models.  The  finance  organization  signs  off  on  any  calculations  of  achieved 
financial  savings  to  ensure  business  and  program  impact. 

5.3  Cross-project  Sharing  of  Best  Practices  and  Lessons  Learned 

Cross-project,  cross-Raytheon,  and  cross-industry  sharing  of  best  practices  and  lessons  learned  is 
integral  to  the  success  of  the  DFSS  program  and  to  our  efforts  for  continuous  improvement.  The 
ability  to  effectively  share  and  collaborate  across  organization  boundaries  is  considered  a  business 
competitive  advantage  integral  to  the  way  we  are  organized,  the  way  we  share,  and  the  way  our 
sharing  success  is  measured. 

Toward  this  end,  the  DFSS  team  is  a  multi-functional  team  that  reports  at  the  engineering  level  as 
part  of  engineering  strategic  process  (not  at  the  systems  or  software  organizational  level),  and  is 
comprised  of  core  team  members  from  each  of  the  Raytheon  engineering  disciplines:  System  Ar¬ 
chitecture,  Design  and  Integration  Directorate  (SADID),  System  Verification,  Test  and  Analysis 
Directorate  (SVTAD),  Software  Engineering  Directorate  (SED),  Electrical  Design  Directorate 
(EDD),  Mechanical  Engineering  Directorate  (MED),  and  Whole  Life  Engineering  Directorate 
(WLED)  (which  includes  reliability,  affordability,  maintainability,  and  so  forth). 

Enterprise-wide  communities  of  practice  (CoP)  also  enable  us  to  share  best  practices  and  lessons 
learned  across  IDS  projects  and  the  Raytheon  Company  as  a  whole.  CoPs  are  focused  groups  of 
SMEs  from  across  Raytheon  that  get  together  regularly  to  advance  a  specific  field  of  study  and 
share  best  practices  and  lessons  learned  across  the  company.  The  DFSS  team  has  initiated  and 
currently  leads  a  number  of  DFSS-related  CoPs,  related  to  such  activities  as  statistical  test  optimi¬ 
zation,  system  cost  modeling,  and  analysis  using  Monte  Carlo  simulation.  The  DFSS  team  is  also 
an  active  participant  in  the  schedule  RAACE  CoP.  All  Raytheon  DFSS  projects  are  tracked 
through  to  completion  and  updated  accordingly  as  new  data  in  the  form  of  actuals,  lessons 
learned,  and  so  forth  that  can  be  obtained  in  our  DFSS  database  management  system. 

Complementing  these  collaborative  sharing  efforts  within  the  company,  the  Raytheon  DFSS  team 
actively  participates  in,  presents  at,  and  leads  industry  conferences,  symposiums,  workshops,  and 
forums  related  to  product  development  optimization.  The  number  of  strategic  industry  technical 
presentations  and  collaborative  knowledge  sharing  events  led  by  or  participated  in  by  the  IDS 
DFSS  team  is  tracked  against  a  developed  annual  goal  as  a  measure  of  our  degree  of  external  en¬ 
gagement,  which  is  a  mechanism  for  further  enabling  our  quest  for  identifying  potential  future 
best  practices,  leveraging  lessons  learned,  and  avoiding  pitfalls  experienced  by  others. 
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Appendix 

Acronym  List 

C5I 

command,  control,  communications,  computing,  cyber,  and  intelligence 

CDM 

combinatorial  design  methods 

CMMI 

Capability  Maturity  Model  Integrated 

CoP 

communities  of  practice 

DFMA 

Design  for  Manufacture  and  Assembly 

DoD 

Department  of  Defense 

DOE 

Design  of  Experiments 

Cl 

configuration  item 

COTS 

commercial -off-the-shelf 

CPI 

cost  performance  index 

CRA 

cost  risk  analysis 

CW 

continuous  wave 

DFSS 

Design  for  Six  Sigma 

EED 

Electrical  Design  Directorate 

EPG 

engineering  process  group 

EW 

electronic  warfare 

GQM 

goal  question  metric 

IPDP 

Integrated  Product  Development  Process 

IPDS 

Integrated  Product  Development  System 

IDS 

Integrated  Defense  Systems 

M&A 

measurement  and  analysis 

MED 

Mechanical  Engineering  Directorate 

NBBE 

non-binding  budgetary  estimate 

NIST 

National  Institute  of  Standards  and  Technology 

OID 

organizational  innovation  and  deployment 

OPM 

Oregon  Productivity  Matrix 
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PPM 


process  performance  model 
QUELCE  Quantifying  Uncertainty  in  Early  Lifecycle  Cost  Estimation 
RAACE  risk  assessment  and  critical  chain  for  execution 

RPAL  Raytheon  process  assets  library 

SADID  System  Architecture,  Design,  and  Integration  Directorate 

SED  Software  Engineering  Directorate 

SLAM  Systems  Lifecycle  Analysis  Model 

SRA  schedule  risk  analysis 

SVTAD  System  Verification,  Test,  and  Analysis  Directorate 

STO  statistical  test  optimization 

WLED  Whole  Life  Engineering  Directorate 
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